{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## K-NN近邻算法\n",
    "\n",
    "K-NN近邻算法或者说K最近邻(kNN，k-NearestNeighbor)分类算法是数据挖掘分类技术中最简单的方法之一。所谓K最近邻，就是k个最近的邻居的意思，说的是每个样本都可以用它最接近的k个邻居来代表。\n",
    "\n",
    "kNN算法的核心思想是如果一个样本在特征空间中的k个最相邻的样本中的大多数属于某一个类别，则该样本也属于这个类别，并具有这个类别上样本的特性。该方法在确定分类决策上只依据最邻近的一个或者几个样本的类别来决定待分样本所属的类别。 kNN方法在类别决策时，只与极少量的相邻样本有关。由于kNN方法主要靠周围有限的邻近的样本，而不是靠判别类域的方法来确定所属类别的，因此对于类域的交叉或重叠较多的待分样本集来说，kNN方法较其他方法更为适合。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### k-Nearest Neighbors（KNN）\n",
    "\n",
    "在本章中，我们将介绍k-Nearest Neighbors（KNN），一个简单的用于分类和回归任务的算法。KNN看上去如此简单，一直很容易小视它的力量。它被广泛应用于现实世界中的各种场景，包括搜索和推荐系统。 下面我们将对KNN与简单的线性回归进行比较和对照，并通过案例来帮助大家理解模型。\n",
    "\n",
    "KNN是回归分类任务的简单模型。它的名称很简单并且直接描述了它的大部分学习算法的精髓。标点邻居是度量空间中训练实例的表示，度量空间是一个特征空间，其中定义了一个集合的所有成员之间的相对距离。 在上一章的住房价格的案例中，我们在度量空间中对训练用实例进行了表示，所有住房价格之间的距离已得到了定义。这些邻近样本数据被用于估计测试实例的对应的值。 超参数k指定在估计中可以使用多少邻居。 超参数是控制算法学习的参数。 超参数不是从训练数据估计而得的，通常需要经过手动进行设置。K值代表的是选择的k个最接近测试实例的k个邻近样本，其邻近成都可以通过一些距离函数测量。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于分类任务而言，训练集由包含特征向量和类标签的一组元组组成。 KNN能够进行二叉、多类和多标签分类，我们稍后将涉及到这些任务，本章将重点介绍二叉分类。 最简单的KNN分类器使用KNN标签的模式对测试实例进行分类。当然，这里可以使用其他策略来达到目标。k通常被设置为奇数以防止half-half困境（某测试样本的K近邻中一半样本是正样本，一半样本是负样本，没法进行该分类。另外一般而言，如果类别个数为偶数，那么K通常会设置为一个奇数；如果类别个数为奇数，K通常设置为偶数，这样就能保证不会有平局的发生）。在回归任务中，特征向量每个都与一个响应变量相关联，该变量采用一个实数标量而不是一个标签。预测是KNN响应变量的平均值或加权平均值。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 惰性学习和非参数模型\n",
    "\n",
    "KNN是一个惰性的机器学习算法，也称为基于实例的学习算法，惰性学习算法在对训练数据集进行存储的时候，对数据只做很少处理或者干脆不做任何处理。 与诸如简单线性回归那用的积极学习算法相反，KNN在训练阶段进行基于训练数据的泛化过程中不对模型进行参数估计。惰性学习算法有优缺点，训练一个积极的学习算法通常需要昂贵的计算开销，但是由此产生的模型的预测往往性价比很高。对于简单的线性回归而言，预测仅需要将学习系数乘以特征，然后再加上训练的到的截距参数即可。而惰性学习算法几乎可以立即展开预测，但预测成本往往可能是昂贵的。比如在最简单的KNN实现中，为了做出预测，需要计算测试实例和所有训练实例之间的距离。\n",
    "\n",
    "与我们后面将要讨论的大多数其他模型相反，KNN是非参数的模型。参数模型使用固定数量的参数或系数，通过总的数据输出，对模型进行定义。参数的数量与训练实例的数量无关。说KNN是非参数模式似乎也并不恰当，因为KNN模型并非没有参数。这么说的原因是非参数意味着模型的参数数量不是固定的，并且可能随培训实例的数量而增长。\n",
    "\n",
    "当训练数据丰富，且我们对响应变量与解释变量之间关系的知识知之甚少的时候，采用KNN这样的非参数模型进行机器学习通常还是很有用的。KNN只有一个假设：彼此靠近的实例可能具有类似的响应变量值。非参数模型所具有的灵活性并不总是分析者所期望的：如果培训数据很少或者您已经对解释变量与响应变量之间的关系有所了解，那么基于假设关系而建构的学习模型可能会更有用。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 应用1：基于KNN的分类\n",
    "\n",
    "这里稍微花点时间回顾一下之前学习到的内容，分类任务的目标是使用一个或多个特征来预测离散响应变量的值。这里我们尝试做一个对玩具进行分类的任务。 假设你需要以一个人的身高和体重为数据来预测他或她的性别。这种类型的问题通常被称为二叉分类，因为响应变量总是非此即彼的两个标签之一。 \n",
    "\n",
    "下表给出了9个训练实例数据：\n",
    "\n",
    "|Height|Weight|Label|\n",
    "|--|--|--|\n",
    "|158 cm|64 kg|male|\n",
    "|170 cm|86 kg|male|\n",
    "|183 cm|84 kg|male|\n",
    "|191 cm|80 kg|male|\n",
    "|155 cm|49 kg|female|\n",
    "|163 cm|59 kg|female|\n",
    "|180 cm|67 kg|female|\n",
    "|158 cm|54 kg|female|\n",
    "|170 cm|67 kg|female|\n",
    "\n",
    "与前面讲过的简单线性回归问题不同，我们现在使用两个解释变量的特征来预测响应变量的值。当然，基于KNN的学习绝对不限于两个特征，该算法可以使用任意数量的特征。但是由于人类认知能力的局限，在对数据进行可视化的时候，没法更有效的显示三个以上的特征。这里我们通过调用matplotlib来创建散点图进行数据的可视化。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XucXHV9//HXW2jQZYGFAtugaIjgBVHQjeAFKVuUWn5W\nQCxgLUW0xkt/sAR7Idso+9Nl1yr+0rVeQalYqgGpeGtLgxhAW0UTCiQgVFnulyCQBZalBNhP/zhn\nwslkZneS3TMzZ8/7+XjMY+ecOZf3fjc5nznf78w5igjMzKy8ntPqAGZm1louBGZmJedCYGZWci4E\nZmYl50JgZlZyLgRmZiXnQmBtRdILJY1L2q6BZRdICknbNyPbtmpmTklfkvTRBpf9mqTBnHKEpH3z\n2LbNPheCNiTpdklvrpr3Hkk/aVWmRkkakHRhjfkNHRgi4s6I6IyIZ/LK0k4kLZX0b1XzflVn3onT\nbS8iPhgRn5ilbC0/mEuaJ+kzku5O3yDcLunvWplpLnIhMGutq4E3VM6AJM0Hfgt4ddW8fdNly2Yp\nsAg4GNgJOBy4tpWB5iIXgoKqfreWPc2XdHj6DuqvJD0g6T5Jx0g6StJ/S3pYUn9m3YMl/VTSWLrs\n5yTNq9rXB9N3pWOSPi9JM8j+HElnSrpV0kOSLpa0W/raZt0okvaRdLWkxyT9MN139bv8d0u6U9KD\nkv4mXe+tQD9wQvpO8vp0/nskjabbu03Su+tk3OY2kbSdpHPSPKPA/5miOX5BcuA/KJ1+E7AKuKVq\n3q0RcW+6/ZdJujz9O94i6fhMrs26e9J/A/dJulfSn9V4l7+rpH9J2+MaSS9O16sUnevT9jtB0u6S\nfpD+vg9L+rGkqY4hR6Vt/aCkT6d/93npuq/MZNxT0oSkPWps47XApRFxbyRuj4ivZ9bdS9I/S/pN\n+vc8LfPav0r6TGZ6haTzp8hbWi4Ec9fvAM8Fng98DDgP+BOgh+TA8lFJ+6TLPgMsAXYHXg8cAXy4\nantvI/lP+SrgeOD3Z5DtVOAY4HeBvYANwOfrLPsN4OfAbwMDwEk1ljkUeGma+2OSXh4RlwFDwEVp\nV9OBknYEPgv8QUTsBLwBuK7OfmfSJu9PX3s1ybvZd9bZBxGxEbgGOCyddRjwY+AnVfOuBkh/h8vT\ndtkTOBH4gqT9q7edFsMzgDeTnFEcXiPCicD/A3YFfg2cneaq7PvAtP0uAj4C3A3sAXSTFNqprlFz\nbPr7vwY4Gnhv+vuuIPm3WPEu4IqI+E2NbfwMOEPShyW9MvsGJC1C3weuJ/l3fgRwuqTK3+G9wEmS\nfi8t+AcDfVPkLa+I8KPNHsDtwDgwlnlMAD/JLBPAvpnprwGD6fPDgSeA7dLpndLlD8ksvwY4ps7+\nTyd5F5bd16GZ6YuBM+usOwBsrMo+ls0L/BI4IrPOfOApYHtgQbrs9sALgaeBjsyyFwIXps8ry74g\n8/rPgRMzWS7MvLZjmuU44Hlb+TdpuE2AHwEfzLx2ZOV3mqLNLk2fXw/sB7y1at7J6fMTgB9Xrf9l\n4Kwa/w7OB4Yzy+1b9Xf4GvCVzOtHATdP8W/s48B3s/OmaK8A3pqZ/jDJwR7gEOBOQOn0auD4OtvZ\nDvhz4D+AJ4F7M21xCHBn1fJLgX/ITB8H3AU8mP17+bH5w2cE7euYiOiqPNjy3eh0HopnB1yfSH+u\nz7z+BNAJIOkl6Sn//ZIeJXknvXvV9u7PPJ+orFvHxdnsaf6sFwGXpl0MYySF4RmSd5lZewEPR8RE\nZt5dNfbXULaIeJzkQPpB4L60S+RltZadYZvsVZXzjlr7yLgaODTtHtsjIn4F/CfJ2MFuwAE8Oz7w\nIuCQStul7fdukjPAatU5trntUp8mOWtYmXb5nDnN71XdBnsBRMQ16b4OT9t/X+B7tTYQEc9ExOcj\n4o1AF8kZy/mSXk7SFntVtUU/m/87+j5JMbklItr+wxat4kJQXBNAR2a61oGgUV8Ebgb2i4idSf4z\nbfMYQAPuIumeyRaL50bEPVXL3QfsJin7e+69FfvZotsiIv49It5CchZyM0mXWS0zaZP7qnK+cJrl\nfwrsQtKl9B9pzkdJ3v2+H7g3Im5Ll70LuKqq7Toj4kN1crwgM701bbeFiHgsIj4SEQuBt5N02Rwx\nxSrVbXBvZvoCku6hk4BLIuJ/Gtj/ExHxeZKuxP1J2uK2qrbYKSKOyqx2NskbjfmS3tXI71lGLgTF\ndR3wx+nA5FtJ+tu31U7Ao8B4+g6t1kFlNn0JOFvSiwAk7SHp6OqFIuIOkm6DgXSQ8fXAH27FftYD\nCyoDmpK6JR2d9rM/SdL9Nlln3Zm0ycXAaZJeIGlXYMp3zhHxBMnveQbJ+EDFT9J52U8L/QB4iaST\nJP1W+nht+g65Vo5TJL08LaYNfb8gYz2wsDIh6W2S9k376R8hOYur134AfylpV0l7k/TNX5R57UKS\nMYQ/Ab5ea+V0n6cr+fDD8yRtL+lkkr/Nf5F0Az4m6a/T17eTdICk16brHgacAvwpcDLw95Kev5Vt\nUAouBMXVR3JQrHQNfGcG2/oL4I+Bx0jeIV809eIzNkLSFbBS0mMkA4KH1Fn23SSDtQ8Bg2m2Jxvc\nz7fSnw9Jupbk3/sZJO9MHyYpnvUO8DNpk/OAfyfp278W+HYD61xFMvib7b74cTpvUyGIiMdIxhxO\nTH+P+4G/BXao3mBE/BvJ4Pgqki6dn6UvNdp+A8AFabfL8SRjFz8kKaA/Bb4QEaumWP+7JGNR1wH/\nAnw1k+0ukrYJNi9+1SaAz5D8ng+SjBccFxGjadfn20g+XXVb+vpXgF0k7UxSYP5vRNwTET9O9/8P\n2QFnS1QGa8wKQdJFJAOaZ7U6S9GkZw3rgB0i4uk2yHM+SbfXslZnKTufEVhbS7s9Xpx+Bv2tJB9D\nnMnZT6lIOlbSDmkX1d8C32+TIrAAeAeZswRrHRcCa3e/A1xJ0h3xWeBDEfFfLU1ULB8AHgBuJenT\nz3v8Z1qSPkFyZvLpzCC4tZC7hszMSs5nBGZmJdfWl++t2H333WPBggWzsq3HH3+cHXfccVa2lZci\nZIRi5CxCRihGziJkhGLkbFbGNWvWPBgRta7htLlWf7W5kUdPT0/MllWrVs3atvJShIwRxchZhIwR\nxchZhIwRxcjZrIzA6vAlJszMbDouBGZmJedCYGZWci4EZmYl50Jgc1JUfT+metrMnuVCYHPOwMAA\nS5Ys2XTwjwiWLFnCwMBAa4OZtSkXAptTIoKxsTFGRkZYsmQJAEuWLGFkZISxsTGfGZjVUIgvlJk1\nShLLly8HYGRkhL333puRkRH6+vpYvnw5vgKx2ZZ8RmBzTrYYVLgImNXnQmBzTmVMICs7ZmBmm8u1\nEEhaIulGSeskfVPScyUNSLpH0nXp46jpt2TWmEoRqHQH9fT00NfXt2nMwMXAbEu5jRGk9wY9Ddg/\nIp6QdDHJ7fUAlkfEOXnt28pLEl1dXZvGBK666qpN3URdXV3uHjKrIe/B4u2B50l6Cuggucfqgpz3\naSU3MDBARGw66FfGDFwEzGrLrWsoIu4BzgHuBO4DHomIlenLp0q6QdL56S30zGZV9UHfRcCsvtzu\nUJYe4P8ZOAEYA74FXAJcDjwIBPAJYH5EvLfG+ouBxQDd3d09K1asmJVc4+PjdHZ2zsq28lKEjFCM\nnEXICMXIWYSMUIyczcrY29u7JiIWTbtgI9eq3pYH8EfAVzPTfwp8oWqZBcC66bbl+xG0pyLkLELG\niGLkzDPj5OTklNNbo+xtmUUb3I/gTuB1kjqUnJcfAfxS0vzMMseS3MTazErKlwRpvTzHCK4h6Qq6\nFlib7utc4FOS1kq6AegFltTfipnNZVF1SZBKEfAlQZor108NRcRZwFlVs0/Kc59mVhzVlwQZGRkB\n8CVBmszfLDazlvIlQVrPhcDMWqrSHZTlb4E3lwuBmbVMdkygr6+PyclJXxKkBXwZajNrmepLgmS7\niXxJkOZxITCzlvIlQVrPXUNm1nK+JEhruRCYmZWcC4GZWcm5EJiZlZwLgZlZybkQmJmVnAuBmVnJ\nuRCYmZWcC4GZWcm5EJiZlZwLgZlZybkQmJmVnAuBmVmbqL7sdrMuw51rIZC0RNKNktZJ+qak50ra\nTdLlkn6V/tw1zwxmZkUwMDCw2T0YKvdqGBgYyH3fuRUCSc8HTgMWRcQBwHbAicCZwBURsR9wRTpt\nZlZaEcHY2NhmN+Sp3LBnbGws9zODvO9HsD3wPElPAR3AvcBS4PD09QuAK4G/zjmHmVnbyt6QZ2Rk\nhJGREYDNbtiT6/7zrDSS+oCzgSeAlRHxbkljEdGVvi5gQ2W6at3FwGKA7u7unhUrVsxKpvHxcTo7\nO2dlW3kpQkYoRs4iZIRi5CxCRihGzqkyrlmzZtPznp6eGe2nt7d3TUQsmnbBiMjlAewK/AjYA/gt\n4DvAnwBjVcttmG5bPT09MVtWrVo1a9vKSxEyRhQjZxEyRhQjZxEyRhQjZ62Mk5OT0dfXF8CmR19f\nX0xOTm7zfoDV0cDxOs/B4jcDt0XEbyLiKeDbwBuA9ZLmA6Q/H8gxg5lZ24vMmEBfXx+Tk5P09fVt\nNmaQpzzHCO4EXiepg6Rr6AhgNfA4cDLwyfTnd3PMYGbW9iTR1dW12ZhAZcygq6sr9zGC3ApBRFwj\n6RLgWuBp4L+Ac4FO4GJJ7wPuAI7PK4OZWVEMDAwQEZsO+pVi0Iz7N+f6qaGIOAs4q2r2kyRnB2Zm\nllF90G9GEQB/s9jMrPRcCMzMSs6FwMys5FwIzMxKzoXAzKzkXAjMzErOhcDMrORcCMzMSs6FwMys\n5FwIzMxKzoXAzKzkXAjMzErOhcDMrORcCMzMSs6FwMys5FwIzMxKzoXAzKzkcrtDmaSXAhdlZi0E\nPgZ0Ae8HfpPO74+If80rh5mZTS3PexbfAhwEIGk74B7gUuAUYHlEnJPXvs3MrHHN6ho6Arg1Iu5o\n0v7MzKxBioj8dyKdD1wbEZ+TNEByVvAIsBr4SERsqLHOYmAxQHd3d8+KFStmJcv4+DidnZ2zsq28\nFCEjFCNnETJCMXIWISMUI2ezMvb29q6JiEXTLhgRuT6AecCDQHc63Q1sR3I2cjZw/nTb6Onpidmy\natWqWdtWXoqQMaIYOYuQMaIYOYuQMaIYOZuVEVgdDRynm9E19AckZwPr08KzPiKeiYhJ4Dzg4CZk\nMDOzOppRCN4FfLMyIWl+5rVjgXVNyGBmZnXk9qkhAEk7Am8BPpCZ/SlJBwEB3F71mpmZNVmuhSAi\nHgd+u2reSXnu08zMto6/WWxmVnIuBGZmJedCYGZWci4EZmYl50JgZlZy035qSNJrasx+BLgjIp6e\n/UhmZtZMjXx89AvAa4AbAAEHADcCu0j6UESszDGfmZnlrJGuoXuBV0fEoojoAV4NjJJ8UexTeYYz\nM7P8NVIIXhIRN1YmIuIm4GURMZpfLDMza5ZGuoZulPRFoHId6BOAmyTtADyVWzIzM2uKRs4I3gP8\nGjg9fYym854CevMKZmZmzdHIGcH+EfEZ4DOVGZLeFhE/AMZzS2ZmZk3RyBnBeZIOqExIOhH4aH6R\nzMysmRo5I3gncImkPwbeBPwpcGSuqczMrGmmLQQRMZqeBXwHuBM4MiKeyD2ZmZk1Rd1CIGktyc1j\nKnYjudfwNZKIiFflHc7MzPI31RnB25qWwszMWqZuIYiIO2ayYUkvBS7KzFoIfAz4ejp/AcmtKo+P\niA0z2ZeZmW273K4+GhG3RMRBEXEQ0ANMAJcCZwJXRMR+wBXptJmZtUizLkN9BHBrepZxNHBBOv8C\n4JgmZTAzsxqaVQhOBL6ZPu+OiPvS5/cD3U3KYGZmNSgipl5Aegfwt8CeJJehFhARsXNDO5DmkVzB\n9BURsV7SWER0ZV7fEBG71lhvMbAYoLu7u2fFihXVi2yT8fFxOjs7Z2VbeSlCRihGziJkhGLkLEJG\nKEbOZmXs7e1dExGLpl0wIqZ8kFxn6OXTLTfF+kcDKzPTtwDz0+fzgVum20ZPT0/MllWrVs3atvJS\nhIwRxchZhIwRxchZhIwRxcjZrIzA6mjgON1I19D6iPjlNpWjxLt4tlsI4HvAyenzk4HvzmDbZmY2\nQ41cYmK1pItIvln8ZGVmRHx7uhUl7UhyA5sPZGZ/ErhY0vuAO4DjtyqxmZnNqkYKwc4kH/3MXl8o\ngGkLQUQ8Dvx21byHSD5FZGZmbaCRaw2d0owgZmbWGlNda+ivIuJTkv6eza85BEBEnJZrMjMza4qp\nzggqA8SrmxHEzMxaY6prDX0//XlBvWXMzKz4mvXNYjMza1MuBDanbdy4sdUR5gy35dw1bSGQ9MZG\n5pm1m+HhYdauXcvw8HCroxSe23Jua+SM4O8bnGfWNoaHhxkcHARgcHDQB7AZcFvOfVN9fPT1wBuA\nPSSdkXlpZ5JbVpq1pcqBa2JiAoCJiYlNB7KlS5e2MlrhuC3LYaqPj84DOtNldsrMfxR4Z56hzLZV\n9YGrwgewree2LI+pPj56FXCVpK/FDG9badYMo6Oj9Pf31319YmKC/v5+TjjhBBYuXNjEZMXjtiyX\nRsYIdpB0rqSVkn5UeeSezGwrLVy4kKGhITo6Omq+3tHRwdDQkA9cDXBblksjF537FvAl4CvAM/nG\nMZuZSldFdZdGR0cHy5Ytc1fGVnBblkcjheDpiPhi7knMZkn2AAY+cM2E27IcpvrU0G7p0+9L+jBw\nKZvfj+DhnLOZbbPsgcoHrplxW859U50RrCG56qjS6b/MvBaAOwetrS1dupSVK1dy5JFHTr+wTclt\nObdN9amhfZoZxCwP8+bNa3WEOcNtOXdNO0Yg6R01Zj8CrI2IB2Y/kpmZNVMjg8XvA14PrEqnDyfp\nNtpH0scj4h/rrSipi+TTRgeQdCe9F/h94P3Ab9LF+iPiX7cpvZmZzVgjhWB74OURsR5AUjfwdeAQ\n4GqgbiEARoDLIuKdkuYBHSSFYHlEnDOj5GZmNisaKQR7V4pA6oF03sOSnqq3kqRdgMOA9wBExEZg\no6R6q5iZWQsoYovbEW++gPQF4IUkXywDOA64m+RTRD+IiN466x0EnAvcBBxI0p3Ul653Csk4w2rg\nIxGxocb6i4HFAN3d3T0rVqzY2t+tpvHxcTo7O2dlW3kpQkYoRs4iZIRi5CxCRihGzmZl7O3tXRMR\ni6ZdMCKmfJB8fPSdwPL08U7SAjLNeouAp4FD0ukR4BNAN8nVS58DnA2cP922enp6YrasWrVq1rZV\nbXJycsrpRuWZcTYVIWcRMkYUI2cRMkYUI2ezMgKrY5rja0RM3zWUbuyS9LE17gbujohr0ulLgDMj\n080k6TzgB1u53bY0MDDA2NgYy5cvRxIRwZIlS+jq6mJgYKDV8czM6qp70TlJP0l/Pibp0czjMUmP\nTrfhiLgfuEvSS9NZRwA3SZqfWexYYN0M8reFiGBsbIyRkRGWLFmyqQiMjIwwNjZWOUMyM2tLU32h\n7ND05071lmnAqcA/pZ8YGiUZG/hsOn4QwO3AB2aw/bYgieXLlwMwMjLCyMgIAH19fZvOEMzM2lVD\nN6+XdKikU9Lnu0tq6FvHEXFdRCyKiFdFxDERsSEiToqIV6bz3h4R983kF2gX2WJQ4SJgZkXQyM3r\nzwL+GqhcaWoecGGeoYqo0h2UVekmMjNrZ42cERwLvB14HCAi7mXzW1eWXnZMoK+vj8nJSfr6+jYb\nMzAza1eNfKFsY0SEpOSzpNKOOWcqHEl0dXVtNiZQ6Sbq6upy95CZtbVGCsHFkr4MdEl6P8n1gs7L\nN1bxDAwMEBGbDvqVYuAiYGbtrpHvEZwj6S3Ao8BLgY9FxOW5Jyug6oO+i4CZFcFUdyg7HfhP4Nr0\nwO+Dv5nZHDTVGcELgL8DXiZpLfAfJIXhP8O3qTQzmzOm+kLZXwCkXwZbBLyB5Ath50oai4j9mxPR\nzMzy1Mhg8fOAnYFd0se9wNo8Q5mZWfNMNUZwLvAK4DHgGpJuof8fNS4ZbWZmxTXVF8peCOwA3A/c\nQ3I10bFmhDIzs+aZaozgrUo+//gKkvGBjwAHSHoY+GlEnNWkjGZmlqMpxwjSexGskzRGckexR4C3\nAQcDLgRmZnPAVGMEp5GcCbwBeIr0o6PA+Xiw2MxszpjqjGAByX2Kl8yVS0WbmdmWphojOKOZQczM\nrDUaujGNmZnNXbkWAkldki6RdLOkX0p6vaTdJF0u6Vfpz13zzGBmZlPL+4xgBLgsIl4GHAj8EjgT\nuCIi9gOuSKfNzKxFcisEknYBDgO+ChARGyNiDDgauCBd7ALgmLwymJnZ9JTXbRQlHQScC9xEcjaw\nBugD7omIrnQZARsq01XrLwYWA3R3d/esWLFiVnKNj4/T2dk5K9vKSxEyQjFyFiEjFCNnETJCMXI2\nK2Nvb++aiFg07YIRkcuD5IqlTwOHpNMjwCeAsarlNky3rZ6enpgtq1atmrVt5aUZGW+99dYZb8Nt\nOXuKkLMIGSOKkbNZGYHV0cDxOs8xgruBuyPimnT6EuA1wHpJ8wHSnw/kmMFqGB4e5sUvfjHDw8Ot\njmJmbSC3QhAR9wN3SXppOusIkm6i7wEnp/NOBr6bVwbb0vDwMIODgwAMDg66GJhZQ/cjmIlTgX9K\nb24zSnJjm+cAF0t6H3AHcHzOGSxVKQITExMATExMbCoKS5cubWU0M2uhXAtBRFxHMlZQ7Yg892tb\nqi4CFS4GZuZvFpfA6Ogo/f39WxSBiomJCfr7+xkdHW1yMjNrBy4EJbBw4UKGhobo6Oio+XpHRwdD\nQ0MsXLiwycnMrB24EJTE0qVLWbZs2RbFoKOjg2XLlrlbyKzEXAhKpLoYuAiYGeT/qSFrM5WDfn9/\nv4uAmQEuBKW0dOlSTjjhBI8JmBngrqHSchEwswoXAjOzknMhMDMrORcCM7OScyEwMys5FwIzs5Jz\nITAzKzkXAjOzknMhMDMrORcCM7OScyEwMys5FwIzs5LLtRBIul3SWknXSVqdzhuQdE867zpJR+WZ\nwczMptaMq4/2RsSDVfOWR8Q5Tdi3mZlNw11DZmYlp4jIb+PSbcAjwDPAlyPiXEkDwCnp/NXARyJi\nQ411FwOLAbq7u3tWrFgxK5nGx8fp7OyclW3Vs3HjRubNm7fN6zcj42woQs4iZIRi5CxCRihGzmZl\n7O3tXRMRi6ZdMCJyewDPT3/uCVwPHAZ0A9uRnI2cDZw/3XZ6enpitqxatWrWtlXL0NBQADE0NLTN\n28g742wpQs4iZIwoRs4iZIwoRs5mZQRWRwPH6ly7hiLinvTnA8ClwMERsT4inomISeA84OA8MzTT\n8PAwg4ODAAwODjI8PNziRGZm08utEEjaUdJOlefAkcA6SfMzix0LrMsrQzNVisDExAQAExMTLgZm\nVgh5fmqoG7hUUmU/34iIyyT9o6SDgABuBz6QY4amqC4CFZViAPgm8WbWtnIrBBExChxYY/5Jee2z\nFUZHR+nv76/7+sTEBP39/b5ZvJm1LX98dIYWLlzI0NAQHR0dNV/v6OhgaGjIRcDM2pYLwSxYunQp\ny5Yt26IYdHR0sGzZMncLmVlbcyGYJdXFwEXAzIqiGZeYKI3KQb+/v99FwMwKw4Vgli1dutQDw2ZW\nKO4ayoGLgJkViQuBmVnJuRCYmZWcC4GZWcm5EJiZlZwLgZlZybkQmJmVnAuBmVnJuRCYmZWcC4GZ\nWcm5EJiZlZwLgZlZyeV60TlJtwOPAc8AT0fEIkm7ARcBC0huVXl8RGzIM4eZmdXXjDOC3og4KCIW\npdNnAldExH7AFem0mZm1SCu6ho4GLkifXwAc04IMZmaWUkTkt3HpNuARkq6hL0fEuZLGIqIrfV3A\nhsp01bqLgcUA3d3dPStWrJiVTI8++ig777zzrGwrL+Pj43R2drY6xrSKkLMIGaEYOYuQEYqRs1kZ\ne3t712R6Y+rK+8Y0h0bEPZL2BC6XdHP2xYgISTUrUUScC5wLsGjRojj88MNnHGZ4eJh58+axcePG\ntr572JVXXsls/L55K0LOImSEYuQsQkYoRs52y5hr11BE3JP+fAC4FDgYWC9pPkD684E8M1QMDw8z\nODgIwODgIMPDw83YrZlZ28utEEjaUdJOlefAkcA64HvAyeliJwPfzStDRaUITExMADAxMeFiYGaW\nyrNrqBu4NBkGYHvgGxFxmaRfABdLeh9wB3B8jhm2KAIVlWIAtHU3kZlZ3nIrBBExChxYY/5DwBF5\n7TdrdHSU/v7+uq9PTEzQ39/vm82bWanN6W8WL1y4kKGhITo6Omq+3tHRwdDQkIuAmZXanC4EkHT7\nLFu2bIti0NHRwbJly9wtZGall/fHR9tC5WBfGRNwETAze1YpCgFsPiDsImBm9qzSFAJIisHKlSs5\n8sgjWx3FzKxtzPkxgmrz5s1rdQQzs7ZSukJgZmabcyEwMys5FwIzs5JzITAzKzkXAjOzknMhMDMr\nuVzvUDZbJP2G5Eqls2F34MFZ2lZeipARipGzCBmhGDmLkBGKkbNZGV8UEXtMt1AhCsFskrS6kVu3\ntVIRMkIxchYhIxQjZxEyQjFytltGdw2ZmZWcC4GZWcmVsRCc2+oADShCRihGziJkhGLkLEJGKEbO\ntspYujECMzPbXBnPCMzMLMOFwMys5OZcIZB0vqQHJK3LzBuQdI+k69LHUen8BZKeyMz/UqsypvNP\nlXSzpBslfSozf6mkX0u6RdLvt1vGVrVjvZySLspkuV3SdZnX2qIt62Vsw7Y8SNLP0iyrJR2cea1d\n2rJmxjZsywMl/VTSWknfl7Rz5rWmt+VmImJOPYDDgNcA6zLzBoC/qLHsguxyLc7YC/wQ2CGd3jP9\nuT9wPbADsA9wK7Bdm2VsSTvWy1n1+meAj7VbW06Rsa3aElgJ/EH6/CjgynZryykytltb/gL43fT5\ne4FPtLIts485d0YQEVcDD7c6x1TqZPwQ8MmIeDJd5oF0/tHAioh4MiJuA34NHEzOtjJjy0z195Yk\n4Hjgm+msdmrLehlbpk7OACrvXHcB7k2ft1Nb1svYMnVyvgS4On1+OXBc+rwlbZk15wrBFE6VdEN6\nyrZrZv7GEfohAAAEcElEQVQ+6WnjVZLe1LJ0yT+SN0m6Js3y2nT+84G7Msvdnc5rhXoZoX3aMetN\nwPqI+FU63U5tWVGdEdqrLU8HPi3pLuAcoHKz73Zqy3oZob3a8kaSgz7AHwF7p89b3pZlKQRfBBYC\nBwH3kZyKkz5/YUQcBJwBfCPbb9dk2wO7Aa8D/hK4OH232E7qZWyndsx6F23wTnsa1RnbrS0/BCyJ\niL2BJcBXW5ilnnoZ260t3wt8WNIaYCdgYwuzbKYUhSAi1kfEMxExCZxHetqVnoo9lD5fQ9I395IW\nxbwb+HYkfg5MklyY6h6efecA8IJ0XivUzNhm7QiApO2BdwAXZWa3U1vWzNiGbXky8O30+bd4tsui\nndqyZsZ2a8uIuDkijoyIHpLif2v6UsvbshSFQNL8zOSxwLp0/h6StkufLwT2A0abnxCA75AMxiLp\nJcA8kqsTfg84UdIOkvZJM/68nTK2WTtWvBm4OSLuzsxrp7aEGhnbsC3vBX43ff57QKULq53asmbG\ndmtLSXumP58DLAMqn2JqfVu2YkQ9zwdJpb0PeIrkHez7gH8E1gI3pI0+P132OJJ+u+uAa4E/bGHG\necCFJEXqWuD3Msv/Dcm7h1tIPx3RThlb1Y71cqbzvwZ8sMbybdGW9TK2W1sChwJrSD7Vcg3Q025t\nWS9jG7ZlH/Df6eOTpFd2aFVbZh++xISZWcmVomvIzMzqcyEwMys5FwIzs5JzITAzKzkXAjOzknMh\nsMKTNF41/R5Jn5tmnbdLOnOaZQ6X9IM6r50uqaPOa1+RtP90uc3ahQuBlVJEfC8iPjmDTZwO1CwE\nEfFnEXHTDLZt1lQuBDanpd8u/WdJv0gfb0znbzprkPTi9Hr2ayUNVp1hdEq6RMk9GP5JidOAvYBV\nklbV2OeVkhalz8clnS3p+nQf3TWW75T0D+n+b5B0XGbdTyu598MPJR2cbntU0ttzaC4rKRcCmwue\np2dvPnId8PHMayPA8oh4Lck3Tb9SY/0RYCQiXknyLdCsV5O8+9+f5MKFb4yIz5Jc1qA3InqnybYj\n8LOIOJDkEsTvr7HMR4FHIuKVEfEq4EeZdX8UEa8AHgMGgbeQXCbl4zW2Y7ZNtm91ALNZ8EQkV5gE\nknf7wKJ08s3A/pkLue4sqbNq/dcDx6TPv0FyKeOKn0d6LaC0yCwAfrIV2TYClXGGNSQH8mpvBk6s\nTETEhsy6l6XP1wJPRsRTktamOcxmhQuBzXXPAV4XEf+TnbkVV/h+MvP8Gbb+/8xT8ex1XLZ2/ey6\nk5UsETGZXrnUbFa4a8jmupXAqZUJSQfVWOZnPHu3qBNrvF7LYyTXlJ8NlwN/XpnQ5jdOMsudC4HN\ndacBi9JB2JuAD9ZY5nTgDEk3APsCjzSw3XOBy2oNFm+DQWBXSeskXU96qW+zZvHVR6300u8DPBER\nIelE4F0RcfR065nNFe5nNIMe4HPpbTfHSG4paFYaPiMwMys5jxGYmZWcC4GZWcm5EJiZlZwLgZlZ\nybkQmJmV3P8CfXNTrMsY+uwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x27a1d767208>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "X_train = np.array([\n",
    "    [158, 64],\n",
    "    [170, 86],\n",
    "    [183, 84],\n",
    "    [191, 80],\n",
    "    [155, 49],\n",
    "    [163, 59],\n",
    "    [180, 67],\n",
    "    [158, 54],\n",
    "    [170, 67]\n",
    "])\n",
    "y_train = ['male', 'male', 'male', 'male', 'female', 'female', 'female', 'female', 'female']\n",
    "\n",
    "plt.figure()\n",
    "plt.title('Human Heights and Weights by Sex')\n",
    "plt.xlabel('Height in cm')\n",
    "plt.ylabel('Weight in kg')\n",
    "\n",
    "for i, x in enumerate(X_train):\n",
    "    plt.scatter(x[0], x[1], c='k', marker='x' if y_train[i] == 'male' else 'D')\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从图中我们可以看到，由x标记表示的男人往往比女性具有更高的身体和更大的体重。这个观察结果可能与大家的经验一致。现在我们来使用KNN来预测一个给定身高和重量数据的个体的性别是男性还是女性。在这里我们先给出一对数据，假设我们想预测一个身高155厘米，体重70公斤的人的性别，首先，我们必须定义预测所需要的距离数据的度量原则。在这个预测场景中，我们建议使用欧氏距离 - 欧几里德空间中的点与点之间的直线距离来作为计算样本距离的计算方法。 二维空间中的欧氏距离的计算公式如下：\n",
    "\n",
    "$ d(p, q) = d(q, p) = \\sqrt( (q_1 - p_1)^2  + ( q_2 - p_2 )^2) $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来，我们必须计算查询实例和所有训练实例之间的距离。\n",
    "\n",
    "我们将k设置为3，即选择最近的三个训练实例来计算预测样本与这三个实例之间的距离作为预测的依据。以下的脚本用来计算测试实例和训练实例之间的距离，并确定最近邻样本所具有的最主要的性别："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  6.70820393,  21.9317122 ,  31.30495168,  37.36308338,\n",
       "        21.        ,  13.60147051,  25.17935662,  16.2788206 ,  15.29705854])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = np.array([[155, 70]])\n",
    "\n",
    "distances = np.sqrt(np.sum((X_train - x)**2, axis=1))\n",
    "distances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['male', 'female', 'female'], \n",
       "      dtype='<U6')"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nearest_neighbor_indices = distances.argsort()[:3]\n",
    "nearest_neighbor_genders = np.take(y_train, nearest_neighbor_indices)\n",
    "nearest_neighbor_genders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'female'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import Counter\n",
    "b = Counter(np.take(y_train, distances.argsort()[:3]))\n",
    "b.most_common(1)[0][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面通过绘制来显示出需要预测性别的样本和与其最近的三个近邻样本的，为了与其他样本相区别，这里使用了放大的标记来进行显示。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X2cHWV99/HPN8FElw2EFLINgoYUfOChRDeAz2UFacpN\nBUQJllJEayr2liXYVkhRcmsgqGi61gcERWmpJkilKrVpkCyoVYEsTUhQqBhQIEhACGRZmpDd3/3H\nzEkmh7O7Z7Nnzjm7832/XvPaM8+/vTaZ35nrmrkuRQRmZlZcExodgJmZNZYTgZlZwTkRmJkVnBOB\nmVnBORGYmRWcE4GZWcE5EVhTkfQySb2SJlax7UxJIWmPesS2u+oZp6QrJX20ym2/LmlxTnGEpIPz\nOLbVnhNBE5L0oKTjy5a9R9KPGxVTtSQtknRdheVVXRgi4jcR0RoR/XnF0kwkXSTpP8qW/XKQZWcM\nd7yI+EBEfKJGsTX8Yi5pkqTPSHo4/YLwoKR/aGRM45ETgVlj/RB4Q+kOSNIM4EXAa8qWHZxuWzQX\nAXOAo4EpwLHAXY0MaDxyIhijyr+tZW/zJR2bfoP6O0mbJD0q6RRJJ0r6H0lPSlqY2fdoST+VtDnd\n9vOSJpWd6wPpt9LNkr4gSaOIfYKkCyX9StLvJF0vaVq6bpdqFEkHSfqhpC2SfpCeu/xb/pmSfiPp\nCUl/n+43F1gIzEu/Sa5Nl79H0ob0eA9IOnOQGHe7TCRNlHRFGs8G4P8MURx3klz4Z6fzbwa6gfvK\nlv0qIjamx3+VpJvTv+N9kk7PxLVLdU/6b+BRSRsl/WWFb/n7SPr3tDxul/QH6X6lpLM2Lb95kvaV\ndFP6+z4p6UeShrqGnJiW9ROSPp3+3Sel+x6RiXG6pD5J+1U4xlHAjRGxMRIPRsQ/ZfbdX9K/Sno8\n/Xuel1n3fUmfycwvk3TNEPEWlhPB+PX7wIuBlwIfA64G/hxoJ7mwfFTSQem2/cACYF/g9cBxwAfL\njncSyX/KPwROB/54FLF9CDgF+CNgf+Ap4AuDbPsN4A7g94BFwFkVtnkT8Mo07o9JenVErAAuA5an\nVU1HStoT+BzwJxExBXgDsGaQ846mTN6frnsNybfZdw5yDiJiG3A78JZ00VuAHwE/Llv2Q4D0d7g5\nLZfpwBnAFyUdWn7sNBleABxPckdxbIUQzgD+H7APcD9waRpX6dxHpuW3HPgw8DCwH9BGkmiH6qPm\n1PT3fy1wMvDe9PddRvJvseTdwC0R8XiFY/wMuEDSByUdkf0Ckiah7wFrSf6dHwecL6n0d3gvcJak\nt6YJ/2igc4h4iysiPDXZBDwI9AKbM1Mf8OPMNgEcnJn/OrA4/Xws8BwwMZ2fkm5/TGb7HuCUQc5/\nPsm3sOy53pSZvx64cJB9FwHbymLfnI0X+AVwXGafGcDzwB7AzHTbPYCXAduBlsy21wHXpZ9L2x6Q\nWX8HcEYmlusy6/ZMYzkNeMkI/yZVlwmwCvhAZt0Jpd9piDK7Mf28FjgEmFu27Oz08zzgR2X7fxm4\npMK/g2uAJZntDi77O3wd+Epm/YnAvUP8G/s48J3ssiHKK4C5mfkPklzsAY4BfgMonV8NnD7IcSYC\nfw38F7AV2Jgpi2OA35RtfxHwtcz8acBDwBPZv5enXSffETSvUyJiamnihd9Gh/O72Nng+lz687HM\n+ueAVgBJr0hv+X8r6RmSb9L7lh3vt5nPfaV9B3F9NvY0/qyXAzemVQybSRJDP8m3zKz9gScjoi+z\n7KEK56sqtoh4luRC+gHg0bRK5FWVth1lmexfFuevK50j44fAm9Lqsf0i4pfAT0jaDqYBh7OzfeDl\nwDGlskvL70ySO8By5XHsdtmlPk1y17AyrfK5cJjfq7wM9geIiNvTcx2blv/BwHcrHSAi+iPiCxHx\nRmAqyR3LNZJeTVIW+5eVxUJ2/Xf0PZJkcl9ENP3DFo3iRDB29QEtmflKF4JqfQm4FzgkIvYi+c+0\n220AVXiIpHommyxeHBGPlG33KDBNUvb3PHAE53lBtUVE/GdEvI3kLuRekiqzSkZTJo+WxfmyYbb/\nKbA3SZXSf6VxPkPy7ff9wMaIeCDd9iHgtrKya42IcweJ44DM/EjK7gUiYktEfDgiZgFvJ6myOW6I\nXcrLYGNm/lqS6qGzgBsi4n+rOP9zEfEFkqrEQ0nK4oGyspgSESdmdruU5IvGDEnvrub3LCIngrFr\nDfBnacPkXJL69t01BXgG6E2/oVW6qNTSlcClkl4OIGk/SSeXbxQRvyapNliUNjK+HvjTEZznMWBm\nqUFTUpukk9N69q0k1W8Dg+w7mjK5HjhP0gGS9gGG/OYcEc+R/J4XkLQPlPw4XZZ9Wugm4BWSzpL0\nonQ6Kv2GXCmOcyS9Ok2mVb1fkPEYMKs0I+kkSQen9fRPk9zFDVZ+AH8raR9JB5LUzS/PrLuOpA3h\nz4F/qrRzes7zlTz88BJJe0g6m+Rv898k1YBbJH0kXT9R0uGSjkr3fQtwDvAXwNnAP0p66QjLoBCc\nCMauTpKLYqlq4N9Gcay/Af4M2ELyDXn50JuPWhdJVcBKSVtIGgSPGWTbM0kaa38HLE5j21rleb6V\n/vydpLtI/r1fQPLN9EmS5DnYBX40ZXI18J8kdft3Ad+uYp/bSBp/s9UXP0qX7UgEEbGFpM3hjPT3\n+C3wSWBy+QEj4j9IGse7Sap0fpauqrb8FgHXptUup5O0XfyAJIH+FPhiRHQPsf93SNqi1gD/Dnw1\nE9tDJGUT7Jr8yvUBnyH5PZ8gaS84LSI2pFWfJ5E8XfVAuv4rwN6S9iJJMP83Ih6JiB+l5/9atsHZ\nEqXGGrMxQdJykgbNSxody1iT3jWsByZHxPYmiOcakmqvixsdS9H5jsCaWlrt8QfpM+hzSR5DHM3d\nT6FIOlXS5LSK6pPA95okCcwE3kHmLsEax4nAmt3vA7eSVEd8Djg3Iv67oRGNLX8FbAJ+RVKnn3f7\nz7AkfYLkzuTTmUZwayBXDZmZFZzvCMzMCq6pu+8t2XfffWPmzJk1P+6zzz7LnnvuWfPj1pJjrA3H\nWBuOsTbqFWNPT88TEVGpD6ddNfrV5mqm9vb2yEN3d3cux60lx1gbjrE2HGNt1CtGYHW4iwkzMxuO\nE4GZWcE5EZiZFZwTgZlZwTkR2LgWZe/JlM+bmROBjWOLFi1iwYIFOy7+EcGCBQtYtGhRYwMzazJO\nBDYuRQSbN2+mq6uLBQsWALBgwQK6urrYvHmz7wzMMsbEC2VmIyWJpUuXAtDV1cWBBx5IV1cXnZ2d\nLF26FPdEbLaT7whs3MomgxInAbMXciKwcavUJpCVbTMws0SuiUDSAkn3SFov6ZuSXixpkaRHJK1J\npxOHP5LZyJSSQKk6qL29nc7Ozh1tBk4GZjvl1kaQjg16HnBoRDwn6XqS4fUAlkbEFXmd20wSU6dO\n3dEmcNttt+2oJpo6daqrh8wy8m4s3gN4iaTngRaSMVZn5nxOMyB5fDQidlz0S20GTgJmu8qtaigi\nHgGuAH4DPAo8HREr09UfknS3pGvSIfTMclF+0XcSMHuh3EYoSy/w/wrMAzYD3wJuAG4GngAC+AQw\nIyLeW2H/+cB8gLa2tvZly5bVPMbe3l5aW1trftxacoy14RhrwzHWRr1i7Ojo6ImIOcNuWE1f1bsz\nAe8CvpqZ/wvgi2XbzATWD3csj0fQ3BxjbRQ9xoGBgSHnq1X0csyiCcYj+A3wOkktSu7HjwN+IWlG\nZptTSQaxNrMCc3cgjZVnG8HtJFVBdwHr0nNdBXxK0jpJdwMdwILBj2Jm412UdQdSSgLuDqR+cn1q\nKCIuAS4pW3xWnuc0s7GlvDuQrq4uAHcHUkd+s9jMGs7dgTSWE4GZNVypOijLb4DXjxOBmTVUtk2g\ns7OTgYEBdwdSZ+6G2swaqrw7kGw1kbsDqQ8nAjNrOHcH0liuGjKzpuDuQBrHicDMrOCcCMzMCs6J\nwMys4JwIzMwKzonAzKzgnAjMzArOicDMrOCcCMzMCs6JwMys4JwIzMwKzonAzKzgnAjMzJpEeZfb\n9eqCO9dEIGmBpHskrZf0TUkvljRN0s2Sfpn+3CfPGMzMxoJFixbtMv5CaZyGRYsW5X7u3BKBpJcC\n5wFzIuJwYCJwBnAhcEtEHALcks6bmRVWRLB58+ZdBuMpDdazefPm3O8M8h6PYA/gJZKeB1qAjcBF\nwLHp+muBW4GP5ByHmVnTyg7G09XVRVdXF8Aug/Xkev48M42kTuBS4DlgZUScKWlzRExN1wt4qjRf\ntu98YD5AW1tb+7Jly2oeX29vL62trTU/bi05xtpwjLXhGGtjqBh7enp2fG5vbx/VeTo6OnoiYs6w\nG0ZELhOwD7AK2A94EfBvwJ8Dm8u2e2q4Y7W3t0ceuru7czluLTnG2nCMteEYa6NSjAMDA9HZ2RnA\njqmzszMGBgZ2+zzA6qjiep1nY/HxwAMR8XhEPA98G3gD8JikGQDpz005xmBm1vQi0ybQ2dnJwMAA\nnZ2du7QZ5CnPNoLfAK+T1EJSNXQcsBp4FjgbuDz9+Z0cYzAza3qSmDp16i5tAqU2g6lTp+beRpBb\nIoiI2yXdANwFbAf+G7gKaAWul/Q+4NfA6XnFYGY2VixatIiI2HHRLyWDeozdnOtTQxFxCXBJ2eKt\nJHcHZmaWUX7Rr0cSAL9ZbGZWeE4EZmYF50RgZlZwTgRmZgXnRGBmVnBOBGZmBedEYGZWcE4EZmYF\n50RgZlZwTgRmZgXnRGBmVnBOBGZmBedEYGZWcE4EZmYF50RgZlZwTgRmZgXnRGBmVnC5jVAm6ZXA\n8syiWcDHgKnA+4HH0+ULI+L7ecVhZmZDy3PM4vuA2QCSJgKPADcC5wBLI+KKvM5tZmbVq1fV0HHA\nryLi13U6n5mZVUkRkf9JpGuAuyLi85IWkdwVPA2sBj4cEU9V2Gc+MB+gra2tfdmyZTWPq7e3l9bW\n1poft5YcY204xtpwjLVRrxg7Ojp6ImLOsBtGRK4TMAl4AmhL59uAiSR3I5cC1wx3jPb29shDd3d3\nLsetJcdYG46xNhxjbdQrRmB1VHGdrkfV0J+Q3A08liaexyKiPyIGgKuBo+sQg5mZDaIeieDdwDdL\nM5JmZNadCqyvQwxmZjaI3J4aApC0J/A24K8yiz8laTYQwINl68zMrM5yTQQR8Szwe2XLzsrznGZm\nNjJ+s9jMrOCcCMzMCs6JwMys4JwIzMwKzonAzKzghn1qSNJrKyx+Gvh1RGyvfUhmZlZP1Tw++kXg\ntcDdgIDDgXuAvSWdGxErc4zPzMxyVk3V0EbgNRExJyLagdcAG0heFPtUnsGZmVn+qkkEr4iIe0oz\nEfFz4FURsSG/sMzMrF6qqRq6R9KXgFI/0POAn0uaDDyfW2RmZlYX1dwRvAe4Hzg/nTaky54HOvIK\nzMzM6qOaO4JDI+IzwGdKCySdFBE3Ab25RWZmZnVRzR3B1ZIOL81IOgP4aH4hmZlZPVVzR/BO4AZJ\nfwa8GfgL4IRco6qD7du309/fT39/PxMnTmx0OGZmDTPsHUH6dNAZwLeB04ATIuLpvAPLw9atW7nu\nuus44ogjmDRpEmvXruVFL3oRRxxxBNdddx1bt25tdIhmZnU3aCKQtE7S3ZLuBm4ApgEHAbeny8aU\nO+64g/33359zzz2X9evX7zJe5/r16zn33HPZf//9ufPOOxsdqplZXQ1VNXRS3aLI2Z133slb3/pW\nnn322UG36e1N2r07Ojro7u7mqKOOqld4ZmYNNWgiiIhfj+bAkl4JLM8smgV8DPindPlMkqEqT4+I\np0ZzrqFs3bqVuXPnDpkEsp599lnmzp3Lxo0bmTx5cl5hmZk1jdx6H42I+yJidkTMBtqBPuBG4ELg\nlog4BLglnc/Nt771LbZt2zaifbZt28YNN9yQU0RmZs2lXt1QHwf8Kr3LOBm4Nl1+LXBKnif+5Cc/\nuaPap1q9vb1cfvnlOUVkZtZc6pUIzgC+mX5ui4hH08+/BdryOml/fz/33HPP8BtWcM8999Df31/j\niMzMmo8iYugNpHcAnwSmk3RDLSAiYq+qTiBNIunB9LCIeEzS5oiYmln/VETsU2G/+cB8gLa2tvZl\ny5aVbzKs/v5+1q5dy2C/4wEHHMDDDz88WNwceeSRDX/HoLe3l9bW1obGMBzHWBuOsTYc404dHR09\nETFn2A2zj1FWmkj6GXr1cNsNsf/JwMrM/H3AjPTzDOC+4Y7R3t4eu2P79u0hKYCK0xVXXDHoOkmx\nffv23TpvLXV3dzc6hGE5xtpwjLXhGHcCVkcV1+lqqoYei4hfVJ2CXujd7KwWAvgucHb6+WzgO6M4\n9pAmTpzIYYcdtlv7HnbYYQ2/GzAzq4dqEsFqScslvVvSO0pTNQeXtCfJADbfziy+HHibpF8Cx6fz\nufnIRz4y4luw1tZWLrww14eZzMyaRjWJYC+SRz9PAP40nap62Swino2I34tMlxQR8buIOC4iDomI\n4yPiyd0JvFrvete7mDRp0oj2mTRpEu985ztzisjMrLkM2+lcRJxTj0DyMnnyZFasWEFHR0dVL5Xt\nueeerFixwi+TmVlhDJoIJP1dRHxK0j+SNKDuIiLOyzWyGjrqqKPo7u5m7ty5bNu2reJ7Ba2trUya\nNIkVK1a4ewkzK5ShqoZKDcSrgZ4K05hy1FFHsXHjRq688koOP/xwJO2YDj/8cK688ko2btzoJGBm\nhTNUX0PfS39eO9g2Y83kyZM588wzOfPMM+nv7+fWW2/l+eef99NBZlZo9XqzuOlMnDhxx2RmVmSF\nTQRWLCPteNAqczmOT8MmAklvrGaZWbNasmQJ69atY8mSJY0OZUxzOY5f1dwR/GOVy8yazpIlS1i8\neDEAixcv9kVsN7kcx7ehHh99PfAGYD9JF2RW7QW4Yt2aXuni1dfXB0BfX9+Oi9lFF13UyNDGFJfj\n+DfUC2WTgNZ0mymZ5c8Afu3Wmlr5xavEF7GRcTkWw1CPj94G3Cbp6zHKYSvN6mnDhg0sXLhw0PV9\nfX0sXLiQefPmMWvWrDpGNra4HIujmjaCyZKukrRS0qrSlHtkZrtp1qxZXHbZZbS0tFRc39LSwmWX\nXeaL1zBcjsUxbF9DwLeAK4GvAB6yy8aEUnVFebVGS0sLF198saszquRyLIZqEsH2iPhS7pGY1Vj2\nIga+eO0ul+P4N9RTQ9PSj9+T9EHgRmBraX3e3Ueb1UL2YuWL1+5zOY5vQ90R9JD0Oqp0/m8z6wJw\nxaCNCRdddBErV67khBNOaHQoY5rLcfwa6qmhg+oZiFmeRjo4kVXmchyfhm0jGGRYyqeBdRGxqfYh\nmZlZPVXTWPw+4PVAdzp/LEm10UGSPh4R/zzYjpKmkjxtdDhJddJ7gT8G3g88nm62MCK+v1vRN4n+\n/n4mTJiApOE3TkUEAwMD7v3UzBqumvcI9gBeHRGnRcRpwKEkF/VjgI8Ms28XsCIiXgUcyc7BbpZG\nxOx0GvNJ4JxzzmHBggVEvGAgt4oiggULFnDOOefQ3+8ncs2ssapJBAdGxGOZ+U3psieB5wfbSdLe\nwFuArwJExLaI2DyaYJvRhAkTmDZtGl1dXVUlg1IS6OrqYtq0aUyY4J7AzayxNNyFS9IXgZeRvFgG\ncBrwMMlTRDdFRMcg+80GrgJ+TnI30AN0pvudQ9LOsBr4cEQ8VWH/+cB8gLa2tvZly5aN9HcbVm9v\nL62trTU51kMPPcSmTZuYPn06Bx544Ki3yyPGvDjG2nCMteEYd+ro6OiJiDnDbhgRQ04kj4++E1ia\nTu8kTSDD7DcH2A4ck853AZ8A2kh6L50AXApcM9yx2tvbIw/d3d01O9bAwEB0dnYGEJ2dndHf37/L\n+v7+/l3WDwwM1D3GvDjG2nCMteEYdwJWxzDX14gYvrE4PdgN6TQSDwMPR8Tt6fwNwIWRqWaSdDVw\n0wiP25QksXTpUgC6urq47bbb6OnpYcKECQwMDNDe3s6aNWvo7Oxk6dKlI2pYNjPL06AV1JJ+nP7c\nIumZzLRF0jPDHTgifgs8JOmV6aLjgJ9LmpHZ7FRg/SjibyqS+OxnP8vs2bNZs2YN7e3tuySB2bNn\n89nPftZJwMyaylAvlL0p/TllsG2q8CHgXyRNAjaQtA18Lm0/COBB4K9GcfymM2HCBHp6enZc/EuP\nh86ePXvHHYKZWTOp5j0CJL0JOCQiviZpX2BKRDww3H4RsYakrSDrrJGHObaUkkH2HQEnATNrVtUM\nXn8JyfsCpV6mJgHX5RnUWFeqDsoqVROZmTWbar6ingq8HXgWICI2suvQlZZR3ibQ39//gjYDM7Nm\nUk0i2JY+OZQ8SyrtmW9IY1dEcMEFF+xIAqXqoJ6enh3J4IILLqj6DWQzs3qoJhFcL+nLwFRJ7wd+\nAFydb1hjT2TeGO7s7NylTaCUDDo7O6t+A9nMrF6qeY/gCklvA54BXgl8LCJuzj2yMaQ8CVR6T2DC\nhAm7vGcA+H0CM2sKQ41Qdj7wE+Cu9MLvi38F1SSBkvKXzsDJwMwab6g7ggOAfwBeJWkd8F8kieEn\n4WEqdxgYGODJJ5+s+o3hbDJ48skn3RW1mTXcUC+U/Q1A+jLYHOANJC+EXSVpc0QcWp8Qm9vEiRP5\n2te+NqLxCErJwEnAzJpBNS+UvQTYC9g7nTYC6/IMaqzZnYu5JCcBM2sKQ7URXAUcBmwBbiepFvps\nVOgy2szMxq6hHh99GTAZ+C3wCElvouNuYBkzs6Ibqo1grpJK78NI2gc+DBwu6UngpxFxSZ1iNDOz\nHA3ZRpC+Ubxe0maSEcWeBk4CjgacCMzMxoGh2gjOI7kTeAPJ2MQ/SadrcGOxmdm4MdQdwUyScYoX\nRMSj9QnHzMzqbag2ggvqGYiZmTWGR0opuO7ubqZPn053d3ejQzGzBsk1EUiaKukGSfdK+oWk10ua\nJulmSb9Mf+6TZww2uO7ubk466SQef/xxTjrpJCcDs4LK+46gC1gREa8CjgR+AVwI3BIRhwC3pPNW\nZ6Uk0NfXB0BfX5+TgVlB5ZYIJO0NvAX4KkBEbIuIzcDJwLXpZtcCp+QVg1VWngRKnAzMikl5DZAi\naTZwFfBzkruBHqATeCQipqbbCHiqNF+2/3xgPkBbW1v7smXLah5jb28vra2tNT9uLdU6xi1btnD/\n/fcPOWTmhAkTOPjgg5kypboRSYtYjnlwjLXhGHfq6OjoiYg5w24YEblMJD2WbgeOSee7gE8Am8u2\ne2q4Y7W3t0ceuru7czluLdUyxlWrVkVLS0tp2NEhp5aWlli1alXdY8yLY6wNx1gb9YoRWB1VXK/z\nbCN4GHg4Im5P528AXgs8JmkGQPpzU44xWGqw6qDBuJrIrDhySwQR8VvgIUmvTBcdR1JN9F3g7HTZ\n2cB38orBEiNNAiVOBmbFkPdTQx8C/kXS3cBs4DLgcuBtkn4JHJ/OW052NwmUOBmYjX/VDEyz2yJi\nDUlbQbnj8jyvJUabBEpKyeCmm26io6OjRtGZWbPwm8Xj2Lx580adBEr6+vqYN29eTY5lZs3FiWAc\nW758OS0tLTU5VktLC8uXL6/JscysuTgRjGMdHR3cdNNNo04GLS0trhYyG8ecCMa50SYDJwGz8c+J\noAB2Nxk4CZgVgxNBQYw0GTgJmBWHE0GBVJsMnATMisWJoGCGSwZOAmbF40RQQIMlAycBs2JyIiio\n8mTgJGBWXE4EBVZKBvvtt5+TgFmB5drXkDW/jo4ONm1yT+BmReY7AjOzgnMiMDMrOCcCM7OCcyIw\nMys4J4IcdXd3M336dI/uZWZNzYkgJ6XRwR5//HEP9WhmTS3XRCDpQUnrJK2RtDpdtkjSI+myNZJO\nzDOGRigfItLj/ppZM6vHHUFHRMyOiOzYxUvTZbMj4vt1iKFuBhsn2MnAzJqVq4ZqaLjB4p0MzKwZ\nKSLyO7j0APA00A98OSKukrQIOCddvhr4cEQ8VWHf+cB8gLa2tvZly5bVPL7e3l5aW1trcqwtW7Zw\n//33MzAwMOy2EyZM4OCDD2bKlCnDblvLGPPiGGvDMdaGY9ypo6Ojp6w2prKIyG0CXpr+nA6sBd4C\ntAETSe5GLgWuGe447e3tkYfu7u6aHGfVqlXR0tISQNVTS0tLrFq1qm4x5skx1oZjrA3HuBOwOqq4\nVudaNRQRj6Q/NwE3AkdHxGMR0R8RA8DVwNF5xpC34aqDBuNqIjNrFrklAkl7SppS+gycAKyXNCOz\n2anA+rxiyNvuJoESJwMzawZ53hG0AT+WtBa4A/j3iFgBfCp9pPRuoANYkGMMuRltEihxMjCzRsut\nG+qI2AAcWWH5WXmds57mzZs36iRQ0tfXx7x589wdtJk1hB8f3U3Lly8fdhD4arW0tLB8+fKaHMvM\nbKScCHbTcIPAV8tDRJpZozkRjMJok4GTgJk1AyeCUdrdZOAkYGbNwomgBkaaDJwEzKyZOBHUSLXJ\nwEnAzJqNE0ENDZcMnATMrBk5EdTYYMnAScDMmpUTQQ7Kk4GTgJk1MyeCnJSSwX777eckYGZNLbcu\nJixJBu42wsyane8IzMwKzonAzKzgnAjMzArOicDMrOCcCMzMCs6JwMys4HJ9fFTSg8AWoB/YHhFz\nJE0DlgMzgQeB0yPiqTzjMDOzwdXjjqAjImZHxJx0/kLglog4BLglnTczswZpRNXQycC16edrgVMa\nEIOZmaUUEfkdXHoAeJqkaujLEXGVpM0RMTVdL+Cp0nzZvvOB+QBtbW3ty5Ytq3l8zzzzDHvttVfN\nj1tLvb29tLa2NjqMITnG2nCMteEYd+ro6OjJ1MYMKu8uJt4UEY9Img7cLOne7MqICEkVM1FEXAVc\nBTBnzpw49thjaxrYkiVLmDRpEtu2beOiiy6q6bFr6dZbb6XWv3utOcbacIy14RhHLteqoYh4JP25\nCbgROBp4TNIMgPRn3TvjWbJkCYsXLwZg8eLFLFmypN4hmJk1jdwSgaQ9JU0pfQZOANYD3wXOTjc7\nG/hOXjFUUkoCfX19APT19TkZmFmh5Vk11AbcmDQDsAfwjYhYIelO4HpJ7wN+DZyeYwy7KE8CJaVk\nADR1NZGxJb87AAAG0klEQVSZWR5ySwQRsQE4ssLy3wHH5XXewWzYsIGFCxcOur6vr4+FCxcyb948\nZs2aVcfIzMwaqzBvFs+aNYvLLrtsyPGEL7vsMicBMyucwiQCSKp9Lr744orjCV988cWuFjKzQirc\nCGWli32pTcBJwMyKrnCJAHZtEHYSMLOiK2QigCQZrFy5khNOOKHRoZiZNVSh2gjKTZo0qdEhmJk1\nXKETgZmZORGYmRWeE4GZWcE5EZiZFZwTgZlZwTkRmJkVXK4jlNWKpMdJeiqttX2BJ3I4bi05xtpw\njLXhGGujXjG+PCL2G26jMZEI8iJpdTXDuDWSY6wNx1gbjrE2mi1GVw2ZmRWcE4GZWcEVPRFc1egA\nquAYa8Mx1oZjrI2mirHQbQRmZuY7AjOzwnMiMDMruHGdCCRdI2mTpPWZZYskPSJpTTqdmC6fKem5\nzPIrGxVjuvxDku6VdI+kT2WWXyTpfkn3SfrjZouxmcpR0vJMHA9KWpNZ1xTlOFiMTVaOsyX9LI1j\ntaSjM+uapRwrxthk5XikpJ9KWifpe5L2yqyreznuIiLG7QS8BXgtsD6zbBHwNxW2nZndrsExdgA/\nACan89PTn4cCa4HJwEHAr4CJTRZj05Rj2frPAB9rtnIcIsamKUdgJfAn6ecTgVubrRyHiLGZyvFO\n4I/Sz+8FPtHIcsxO4/qOICJ+CDzZ6DiGMkiM5wKXR8TWdJtN6fKTgWURsTUiHgDuB44mZyOMsSGG\n+ltLEnA68M10UTOV42AxNsQgMQZQ+va6N7Ax/dxM5ThYjA0xSIyvAH6Yfr4ZOC393JByzBrXiWAI\nH5J0d3r7tk9m+UHp7eNtkt7csOiSfzBvlnR7GstR6fKXAg9ltns4XdYIg8UIzVOOJW8GHouIX6bz\nzVSOJeUxQvOU4/nApyU9BFwBlAb5bqZyHCxGaJ5yvIfkog/wLuDA9HPDy7GIieBLwCxgNvAoye04\n6eeXRcRs4ALgG9k6vDrbA5gGvA74W+D69BtjMxksxmYqx5J30+Bv2lUoj7GZyvFcYEFEHAgsAL7a\noDiGMliMzVSO7wU+KKkHmAJsa1AcL1C4RBARj0VEf0QMAFeT3oKlt2W/Sz/3kNTTvaJBYT4MfDsS\ndwADJJ1UPcLObxEAB6TLGqFijE1WjkjaA3gHsDyzuJnKsWKMTVaOZwPfTj9/i53VFs1UjhVjbKZy\njIh7I+KEiGgnSfq/Slc1vBwLlwgkzcjMngqsT5fvJ2li+nkWcAiwof4RAvBvJI2xSHoFMImkp8Lv\nAmdImizpoDTGO5opxiYrR4DjgXsj4uHMsmYqR6gQY5OV40bgj9LPbwVK1VfNVI4VY2ymcpQ0Pf05\nAbgYKD3B1PhyrHdrej0nkqz7KPA8yTfY9wH/DKwD7k7/ADPSbU8jqcNbA9wF/GkDY5wEXEeSpO4C\n3prZ/u9JvkncR/qURDPF2EzlmC7/OvCBCts3RTkOFmMzlSPwJqCH5MmW24H2ZivHwWJssnLsBP4n\nnS4n7dmhUeWYndzFhJlZwRWuasjMzHblRGBmVnBOBGZmBedEYGZWcE4EZmYF50RgY56k3rL590j6\n/DD7vF3ShcNsc6ykmwZZd76klkHWfUXSocPFbdYsnAiskCLiuxFx+SgOcT5QMRFExF9GxM9HcWyz\nunIisHEtfbP0XyXdmU5vTJfvuGuQ9AdpX/brJC0uu8NolXSDknEX/kWJ84D9gW5J3RXOeaukOenn\nXkmXSlqbnqOtwvatkr6Wnv9uSadl9v20kvEefiDp6PTYGyS9PYfisoJyIrDx4CXaOfDIGuDjmXVd\nwNKIOIrkLdOvVNi/C+iKiCNI3gLNeg3Jt/9DSTorfGNEfI6kS4OOiOgYJrY9gZ9FxJEkXRC/v8I2\nHwWejogjIuIPgVWZfVdFxGHAFmAx8DaSrlE+XuE4Zrtlj0YHYFYDz0XSuySQfNsH5qSzxwOHZjpv\n3UtSa9n+rwdOST9/g6Qb45I7Iu0DKE0yM4EfjyC2bUCpnaGH5EJe7njgjNJMRDyV2XdF+nkdsDUi\nnpe0Lo3DrCacCGy8mwC8LiL+N7twBL16b8187mfk/2eej539uIx0/+y+A6VYImIg7bHUrCZcNWTj\n3UrgQ6UZSbMrbPMzdo4WdUaF9ZVsIelTvhZuBv66NKNdB0syy50TgY135wFz0kbYnwMfqLDN+cAF\nku4GDgaeruK4VwErKjUW74bFwD6S1ktaS9q9t1m9uPdRK7z0fYDnIiIknQG8OyJOHm4/s/HC9Yxm\n0A58Ph1qczPJkIJmheE7AjOzgnMbgZlZwTkRmJkVnBOBmVnBORGYmRWcE4GZWcH9f8d9xZLiOktV\nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x27a1fa808d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot for fig2\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "X_train = np.array([\n",
    "    [158, 64],\n",
    "    [170, 86],\n",
    "    [183, 84],\n",
    "    [191, 80],\n",
    "    [155, 49],\n",
    "    [163, 59],\n",
    "    [180, 67],\n",
    "    [158, 54],\n",
    "    [170, 67]\n",
    "])\n",
    "y_train = ['male', 'male', 'male', 'male', 'female', 'female', 'female', 'female', 'female']\n",
    "\n",
    "plt.figure()\n",
    "plt.title('Human Heights and Weights by Sex')\n",
    "plt.xlabel('Height in cm')\n",
    "plt.ylabel('Weight in kg')\n",
    "\n",
    "for i, x in enumerate(X_train):\n",
    "    plt.scatter(x[0], x[1], c='k', marker='x' if y_train[i] == 'male' else 'D')\n",
    "plt.scatter(158, 64, s=200, c='k', marker='x')\n",
    "plt.scatter(163, 59, s=200, c='k', marker='D')\n",
    "plt.scatter(158, 54, s=200, c='k', marker='D')\n",
    "plt.scatter(155, 70, s=200, c='k', marker='o')\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "三个与预测样本最为邻近的样本，其中两个是女性，一个是男性。 因此，我们可以判断预测样本是女性。\n",
    "\n",
    "现在我们使用scikit-learn来实现一个KNN分类器："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import LabelBinarizer\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "lb = LabelBinarizer()\n",
    "y_train_binarized = lb.fit_transform(y_train)\n",
    "y_train_binarized"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['female'], \n",
       "      dtype='<U6')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "K = 3\n",
    "clf = KNeighborsClassifier(n_neighbors=K)\n",
    "clf.fit(X_train, y_train_binarized.reshape(-1))\n",
    "prediction_binarized = clf.predict(np.array([155, 70]).reshape(1, -1))[0]\n",
    "predicted_label = lb.inverse_transform(prediction_binarized)\n",
    "predicted_label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "本案例中用到的标签是字符串。我们首先使用`LabelBinarizer`将它们转换为整数。`LabelBinarizer`实现了数据类型转换接口，它涉及到`fit`，`transform`和`fit_transform`等方法。`fit`方法为数据类型转换提供了准备，实际上它实现的是从标签字符串到整数的映射。`transform`方法将转换映射对输入标签进行了操作。`fit_transform`方法是一种方便的调用`fit`和`transform`的方法。数据类型转换应仅适用于训练集。如果单独进行训练和测试集的拟合和转换，可能导致从标签到整数的映射不一致的情况发生。也就是说，在本案例中，不一致可能表现为男性可能会在训练集中映射为1，而在测试集中映射为0。另外，还应避免对整个数据集进行拟合，因为对于某些数据类型转换，有可能会将有关测试集的信息泄漏给模型，导致诸如过拟合现象的发生，导致基于测试集的预测性能测量可能非常乐观，但事实上其泛化能力可能存在问题。降低预测模型应对实际需要的能力。在后面的内容中讲到从文本中提取特征的时侯我们会更多地讨论这个值得注意的隐患。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面，我们来对`KNeighborsClassifier`进行初始化。 即使KNN是一个惰性的学习算法，也需要进行估计器的实现。我们调用`fit`和`predict`的方式跟我们之前进行简单线性回归时一样。在学习的最后阶段，我们可以使用我们拟合出的`LabelBinarizer`来反转转换并将整数映射为一个字符串标签返回值。 现在我们来使用我们的分类器对测试数据进行预测，并对这个预测器的性能进行评估。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 测试数据\n",
    "\n",
    "|Height|Weight|Label|\n",
    "|--|--|--|\n",
    "|168 cm|65 kg|male|\n",
    "|180 cm|96 kg|male|\n",
    "|160 cm|52 kg|female|\n",
    "|169 cm|67 kg|female|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Binarized labels: [1 1 0 0]\n",
      "Binarized predictions: [0 1 0 0]\n",
      "Predicted labels: ['female' 'male' 'female' 'female']\n"
     ]
    }
   ],
   "source": [
    "X_test = np.array([\n",
    "    [168, 65],\n",
    "    [180, 96],\n",
    "    [160, 52],\n",
    "    [169, 67]\n",
    "])\n",
    "y_test = ['male', 'male', 'female', 'female']\n",
    "y_test_binarized = lb.transform(y_test)\n",
    "print('Binarized labels: %s' % y_test_binarized.T[0])\n",
    "\n",
    "predictions_binarized = clf.predict(X_test)\n",
    "print('Binarized predictions: %s' % predictions_binarized)\n",
    "print('Predicted labels: %s' % lb.inverse_transform(predictions_binarized))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过将我们的测试标签与我们的分类器的预测进行比较，我们发现它的错误地预测了其中一个测试样本，预测为男性，而测试实例是女性。这里稍微回顾一下前一章的内容，我们在机器学习基础中提到，二叉分类任务可能会出现两种类型的错误：假阳性和假阴性。机器学习中有许多性能指标用于对分类器进行评估，根据应用中错误类型的后果来采取合适的评估方式非常重要，不同的评估方法有不同的适用环境。下面，我们将使用几种常见的性能指标来评估我们刚刚建构的分类器，这些指标包括准度、精度和召回率。\n",
    "\n",
    "### 准度\n",
    "\n",
    "准度是正确分类的测试实例的比例，我们刚才建构的这个预测模型在测试集上的表现为四个样本有一个错误预测，所以准度是75％。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.75\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "print('Accuracy: %s' % accuracy_score(y_test_binarized, predictions_binarized))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 精度\n",
    "\n",
    "精度是预测为真正为阳性的测试实例的比例。在本例中，阳性的类别是男性。 男，女分属于阳性还是阴性，这个分类在这里是任意分配的，完全可以对换。 我们的分类器预测出其中一个测试实例是阳性的类，这个实例正好也是真正的阳性类，所以分类器的精度是100％。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Precision: 1.0\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import precision_score\n",
    "print('Precision: %s' % precision_score(y_test_binarized, predictions_binarized))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 召回率\n",
    "\n",
    "召回率是指预测为阳性样本占的真阳性样本的比例。我们的分类器预测出了两个真阳性样本中的一个，因此其召回率为50％。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recall: 0.5\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import recall_score\n",
    "print('Recall: %s' % recall_score(y_test_binarized, predictions_binarized))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### F1度量\n",
    "\n",
    "这里我们还可以通过一个称为__F1分值__或__F1度量__的统计量来评价预测模型的性能，__F1__是对精度和召回率的总评， __F1度量__是精度和召回的调和平均数（harmonic mean），又称倒数平均数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F1 score: 0.666666666667\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import f1_score\n",
    "print('F1 score: %s' % f1_score(y_test_binarized, predictions_binarized))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "值得注意的是，精度和召回率的算术平均值是__F1__分值的上限。 __F1__分值会随着精度和召回率之间的差异而增加，从而惩罚分类器。另外，__ Matthews correlation coefficient马修斯相关系数（MCC）__是衡量二进制分类器性能的__F1分数__的替代方法（__MCC__是表征数据对某一特征分类的关联性的，其值在[-1，1]，-1代表完全相悖的预测；1代表完全正确的预测；0代表随机预测）。一个完美的分类器的__MCC__是1。一个简单的分类器，随机地预测得分0，一个完全错误的分类器将得分-1。即使测试集中样本的类别分布的比例严重不平衡，__MCC__也是有用的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Matthews correlation coefficient: 0.57735026919\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import matthews_corrcoef\n",
    "print('Matthews correlation coefficient: %s' % matthews_corrcoef(y_test_binarized, predictions_binarized))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "scikit-learn中提供了一个非常方面的函数__classification_report__来输出__精度__、__召回率__和__F1值__。下面是调用方法。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "       male       1.00      0.50      0.67         2\n",
      "\n",
      "avg / total       1.00      0.50      0.67         2\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "print(classification_report(y_test_binarized, predictions_binarized, target_names=['male'], labels=[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 应用2：基于KNN的回归"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来，我们试着用KNN完成一个回归任务。在这里，我们将利用一个个体的身高和性别数据对其体重进行预测。下面列出来训练数据集和测试数据集。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练数据集\n",
    "\n",
    "|Height|Sex|Weight|\n",
    "|--|--|--|\n",
    "|158 cm|male|64 kg|\n",
    "|170 cm|male|66 kg|\n",
    "|183 cm|male|84 kg|\n",
    "|191 cm|male|80 kg|\n",
    "|155 cm|female|49 kg|\n",
    "|163 cm|female|59 kg|\n",
    "|180 cm|female|67 kg|\n",
    "|158 cm|female|54 kg|\n",
    "|170 cm|female|67 kg|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 测试数据集\n",
    "\n",
    "|Height|Sex|Weight|\n",
    "|--|--|--|\n",
    "|160 cm|male|66 kg|\n",
    "|196 cm|male|87 kg|\n",
    "|168 cm|female|68 kg|\n",
    "|177 cm|female|74 kg|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们将实例化并拟合`KNeighborsRegressor`，并使用它来预测体重。这个数据集里的性别特征已被编码为二进制值。值得注意的是，性别特征的取值范围是从`0`到`1`，而样本身高特征的取值范围则是从`155`到`191`。我们将在下一节讨论为什么这是一个问题，以及如何改进。 在房屋价格的案例中，我们使用决定系数R^2来衡量我们的模型的表现。这个例子中，也会将使用它来再次测量回归函数的性能，并还会对回归性能的评价引入平均绝对误差（__MAE__）和均方误差（__MSE__）这两个性能指标，这样我们就可以接触到更为丰富多样的评价指标。以备不时之需。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  7.,  18.,  36.,  29.])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.neighbors import KNeighborsRegressor\n",
    "\n",
    "X_train = np.array([\n",
    "    [158, 64, 1],\n",
    "    [170, 86, 1],\n",
    "    [183, 84, 1],\n",
    "    [191, 80, 1],\n",
    "    [155, 49, 0],\n",
    "    [163, 59, 0],\n",
    "    [180, 67, 0],\n",
    "    [158, 54, 0],\n",
    "    [170, 67, 0]\n",
    "])\n",
    "y_train = [7, 12, 29, 18, 11, 16, 29, 22, 36]\n",
    "\n",
    "X_test = np.array([\n",
    "    [160, 66, 1],\n",
    "    [196, 87, 1],\n",
    "    [168, 68, 0],\n",
    "    [177, 74, 0]\n",
    "])\n",
    "y_test = [9, 13, 26, 21]\n",
    "\n",
    "K = 1\n",
    "clf = KNeighborsRegressor(n_neighbors=K)\n",
    "clf.fit(X_train, y_train)\n",
    "predictions = clf.predict(np.array(X_test))\n",
    "predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-0.0919377652051\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import r2_score\n",
    "print(r2_score(y_test, predictions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicted weights: [ 59.          77.          70.66666667  72.66666667]\n",
      "Actual weights: [66, 87, 68, 74]\n",
      "Coefficient of determination: 0.412506459948\n",
      "Mean absolute error: 5.25\n",
      "Mean squared error: 39.4722222222\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.neighbors import KNeighborsRegressor\n",
    "from sklearn.metrics import mean_absolute_error, mean_squared_error,r2_score\n",
    "\n",
    "X_train = np.array([\n",
    "    [158, 1],\n",
    "    [170, 1],\n",
    "    [183, 1],\n",
    "    [191, 1],\n",
    "    [155, 0],\n",
    "    [163, 0],\n",
    "    [180, 0],\n",
    "    [158, 0],\n",
    "    [170, 0]\n",
    "])\n",
    "y_train = [64, 86, 84, 80, 49, 59, 67, 54, 67]\n",
    "\n",
    "X_test = np.array([\n",
    "    [160, 1],\n",
    "    [196, 1],\n",
    "    [168, 0],\n",
    "    [177, 0]\n",
    "])\n",
    "y_test = [66, 87, 68, 74]\n",
    "\n",
    "K = 3\n",
    "clf = KNeighborsRegressor(n_neighbors=K)\n",
    "clf.fit(X_train, y_train)\n",
    "predictions = clf.predict(np.array(X_test))\n",
    "\n",
    "print('Predicted weights: %s' % predictions)\n",
    "print('Actual weights: %s' % y_test)\n",
    "\n",
    "print('Coefficient of determination: %s' % r2_score(y_test,predictions))\n",
    "print('Mean absolute error: %s' % mean_absolute_error(y_test,predictions))\n",
    "print('Mean squared error: %s' % mean_squared_error(y_test,predictions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.412506459948\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import r2_score\n",
    "print(r2_score(y_test, predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MAE和MSE/MSD\n",
    "\n",
    "__MAE__是预测误差绝对值的平方和，__MAE__的计算公式如下。\n",
    "\n",
    "$ MAE = \\frac{1}{n}\\sum_{i=0}^{n-1}{|{y_i}-{\\hat{y}_i}|} $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__MSE__，或者均方差__MSD__，是一个比__MAE__更为普及的度量指标。计算公式如下。__MSE__是预测误差平方和的平均值。\n",
    "\n",
    "$ MSE = \\frac{1}{n}\\sum_{i=0}^{n-1}{({y_i}-{\\hat{y}_i})^2} $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "回归的性能评价标准不对误差的方向加以考虑是重要的,否则回归带来的预测误差的正负值会相互抵消,是的性能评价不符合实际情况。 __MSE__和__MAE__分别通过平均误差和绝对值来实现。__MSE__惩罚异常值的强度要超过__MAE__;将一个大误差平方化使它对总误差的贡献不成比例。这在某些问题中可能是可取的，但__MSE__指标在回归性能的评估指标中通常是首选的，因为__MSE__具有有用的数学性质。这里再次提醒一下，普通线性回归（如前一章简单线性回归问题）的思想就是为了使__MSE__的平方根最小化而进行拟合。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 特征缩放\n",
    "\n",
    "当特征值取值相近时，许多学习算法通常会有更好的表现。比如在之前的案例中，我们使用了两个特征：一个代表人的性别的二叉特征（取值范围[0,1]）和一个连续值的特征，代表人的身高（cm）。设想一下，有一个数据集，我们有一个170厘米高的男人和一个160厘米高的女人。对一个身高164厘米高的个体而言，这个个体是更接近女性还是男性？对于体重预测案例而言，我们可能更加偏向于这个个体为男性多一些.在预测体重的时候，高度特征上6厘米的差异所具有的影响力应该小于性别的差异所导致的影响。另外，如果我们测定高度的数据采用不同的单位，比如米、厘米甚至毫米，还是致使数据发生很大的改变，导致以距离测量为判断依据的KNN算法出现很大的偏差。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "60.0\n",
      "40.0124980475\n",
      "162.3\n",
      "162.40307878855006\n"
     ]
    }
   ],
   "source": [
    "from scipy.spatial.distance import euclidean\n",
    "\n",
    "# heights in millimeters\n",
    "X_train = np.array([[1700, 1],[1600, 0]])\n",
    "\n",
    "x_test = np.array([1640, 1]).reshape(1, -1)\n",
    "\n",
    "print(euclidean(X_train[0, :], x_test))\n",
    "print(euclidean(X_train[1, :], x_test))\n",
    "\n",
    "# heights in meters\n",
    "X_train = np.array([[1.7, 1],[1.6, 0]])\n",
    "\n",
    "x_test = np.array([164, 1]).reshape(1, -1)\n",
    "\n",
    "print(euclidean(X_train[0, :], x_test))\n",
    "print(euclidean(X_train[1, :], x_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`scikit-learn`的`standardscaler`是个用了对数据的尺度进行标准化的转换接口，通过对特征进行尺度调节使特征数据为单位方差。`standardscaler`首先通过从每个实例的特征值中减去每个特征的平均值来进行特征标定。然后通过将每个实例的值除以特征的标准偏差来对特征进行缩放。这使得数据标准化为具有零均值和单位方差。像之前接触到的`LabelBinarizer`一样，`standardscaler`实现数据转换。\n",
    "\n",
    "这里我们可以对刚才的那个案例中的数据进行规范化处理，再进行拟合和回归，同时也比较两种模型的性能。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 1.7  1. ]\n",
      " [ 1.6  0. ]]\n",
      "[[ 1.  1.]\n",
      " [-1. -1.]]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\lymanZHANG\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\validation.py:429: DataConversionWarning: Data with input dtype int32 was converted to float64 by StandardScaler.\n",
      "  warnings.warn(msg, _DataConversionWarning)\n"
     ]
    },
    {
     "ename": "ValueError",
     "evalue": "Found input variables with inconsistent numbers of samples: [2, 9]",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-25-8f2e45ae826c>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      9\u001b[0m \u001b[0mX_test_scaled\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mss\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtransform\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX_test\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     10\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 11\u001b[1;33m \u001b[0mclf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX_train_scaled\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0my_train\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     12\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     13\u001b[0m \u001b[0mpredictions\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mclf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpredict\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX_test_scaled\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mC:\\Users\\lymanZHANG\\Anaconda3\\lib\\site-packages\\sklearn\\neighbors\\base.py\u001b[0m in \u001b[0;36mfit\u001b[1;34m(self, X, y)\u001b[0m\n\u001b[0;32m    739\u001b[0m         \"\"\"\n\u001b[0;32m    740\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m(\u001b[0m\u001b[0mKDTree\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mBallTree\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 741\u001b[1;33m             \u001b[0mX\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0my\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mcheck_X_y\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0my\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m\"csr\"\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmulti_output\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    742\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_y\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0my\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    743\u001b[0m         \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_fit\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mC:\\Users\\lymanZHANG\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\validation.py\u001b[0m in \u001b[0;36mcheck_X_y\u001b[1;34m(X, y, accept_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, warn_on_dtype, estimator)\u001b[0m\n\u001b[0;32m    529\u001b[0m         \u001b[0my\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0my\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mastype\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfloat64\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    530\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 531\u001b[1;33m     \u001b[0mcheck_consistent_length\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0my\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    532\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    533\u001b[0m     \u001b[1;32mreturn\u001b[0m \u001b[0mX\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0my\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mC:\\Users\\lymanZHANG\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\validation.py\u001b[0m in \u001b[0;36mcheck_consistent_length\u001b[1;34m(*arrays)\u001b[0m\n\u001b[0;32m    179\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0muniques\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;33m>\u001b[0m \u001b[1;36m1\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    180\u001b[0m         raise ValueError(\"Found input variables with inconsistent numbers of\"\n\u001b[1;32m--> 181\u001b[1;33m                          \" samples: %r\" % [int(l) for l in lengths])\n\u001b[0m\u001b[0;32m    182\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    183\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: Found input variables with inconsistent numbers of samples: [2, 9]"
     ]
    }
   ],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "ss = StandardScaler()\n",
    "\n",
    "X_train_scaled = ss.fit_transform(X_train)\n",
    "\n",
    "print(X_train)\n",
    "print(X_train_scaled)\n",
    "\n",
    "X_test_scaled = ss.transform(X_test)\n",
    "\n",
    "clf.fit(X_train_scaled, y_train)\n",
    "\n",
    "predictions = clf.predict(X_test_scaled)\n",
    "\n",
    "print('Predicted wieghts: %s' % predictions)\n",
    "print('Coefficient of determination: %s' % r2_score(y_test,predictions))\n",
    "print('Mean absolute error: %s' % mean_absolute_error(y_test,predictions))\n",
    "print('Mean squared error: %s' % mean_squared_error(y_test,predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看出，数据经过标准化后，我们的模型表现要好很多。性别特征对模型的预测施加的影响更大，有助于模型做出更好的预测。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 单元知识总结\n",
    "\n",
    "本知识单元中引入了__KNN__，一个简单但是非常强大的机器学习算法，可以用于实现一些分类和回归任务。__KNN__是一种惰性学习算法，属于非参数模型。该算法不是基于固定的参数在学习训练集数据的基础上进行预测，而是在对所有实例样本与预测样本进行距离计算的基础上，找出与预测样本最为邻近的k个样本，将这k个样本的相关属性作为目标样本的预测属性。本单元通过相关案例进行了基于__KNN__的分类和回归实践，并引入了__scikit-learn__中的数据转换接口，比如调用`LabelBinarizer`将文本串转换为二叉标签，调用`StandardScaler`进行数据的标准化以提高模型的表现。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
